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SUMMARY 


Let  X  be  an  n— cllmenelonal  vector  describing  the 
state  of  a  physical  system,  and  let  x  satisfy  an  equation 
of  the  form 

^  -  Ax  +  V,  x(o)  -  c. 

The  control  vector  v  is  to  be  chosen,  subject  to  various 
constraints,  to  minimize  or  maximize  a  given  function  of 
the  first  k  components  of  x  at  time  T.  This  Is  often  called 
a  terminal  control  process. 

A  straightforward  application  of  the  functional 
equation  techniques  of  dynamic  programming  leads  to  the 
computation  of  a  sequence  of  functions  of  dimension  n,  the 
dimension  of  the  state  vector  n.  Consequently,  the  prob¬ 
lem  cannot,  at  the  present  time,  be  resolved  computationally 
In  any  routine  fashion  If  n  exceeds  2. 

By  means  of  a  preliminary  transformation,  based 
upon  the  superposition  principle  of  linear  systems,  the 
functional  equation  technique  can  be  applied  In  such  a  way 
as  to  make  the  problem  depend  upon  sequences  of  functions 
of  dimension  k.  In  a  number  of  Important  applications  In 
economics  and  engineering,  k  Is  equal  to  one  or  two,  which 
means  that  the  variational  problems  arising  In  determining 
optimal  behavior  can  be  resolved  regardless  of  the 
dimensionality  of  the  underlying  system. 

In  particular,  problems  of  this  nature  Involving 
time-lags  and  other  hereditary  effects  can  now  be  treated 
by  the  functional  equation  technique. 
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T«rmlnal  Control,  Tlm«— Lags  and  Dynamic  Programming 

By 

Richard  Bellman 

Tha  RAND  Corporation,  Santa  Monica,  California 


^1.  Introduction .  A  type  of  control  process  that  Is  common  to 
economic  and  engineering  fields  is  that  of  maximizing  a  function¬ 
al  of  the  form 

i  1 

(1.1) 


T  T 

J(v)  -  y  h(x)da(t)  +  J'  lc(v)dt, 
0  0 


over  all  vector  functions  v(t)  satisfying  constraints  of  the 
form 

<  0,  1  “  1,2, ••.,!<,  (1.2) 

where  x(t)  Is  in  n-dlmenslonal  vector  function  determined  by  the 
differential  equation 


dx 


-  g(x,v),  x(0)  -  c. 


do) 


If  0(t)  la  a  step  function  with  a  Jump  at  T,  we  have  what  Is 
often  called  a  terminal  control  process. 

Using  the  functional  equation  approach  of  dynamic  pro¬ 
gramming  we  write 

f(c,T)  -  Min  J(v),  (1,4) 

V 

and  derive  a  functional  equat'.on  for  f(c,T)  having  the  form 

f(c,T)  -  Min  [f  (c  +  8g(c,v(0)J,  T  -  sjl  o(s). 
v(0)  •-  J 

This  may  be  used  either  In  discrete  form  for  computational  pur¬ 
poses,  or  In  the  limiting  continuous  version  for  analytic  and 
computational  purposes.  The  vector  v(o)  satisfies  the  con-, 
Btralnts  Rj[c,v(0)]  <  0,  derived  from  (1.2);  See  for 
further  discussion  and  applications. 
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Du0  to  the  limited  capacity  of  present  day  digital 
computers,  Me  encounter  grave  computational  difficulties  Mhen 
the  dlmenaion  of  x  exceeds  two.  In  this  note  ms  wish  to  dis¬ 
cuss  terminal  control  problems  for  linear  processes,  where 
the  equation  corresponding  to  (1.3)  Is 

^  -  Ax  +  V,  x(0)  -  c.  (1,6) 

By  means  of  a  preliminary  transformation,  a  number  of  Important 
problems  can  be  treated  by  means  of  the  functional  equation 
approach  of  dynamic  programming  In  terms  of  functions  of  many 
fewer  dimensions.  In  particular,  we  will  be  able  to  treat 
control  problems  Involving  time-lags  where  the  underlying 
equations  are  differential— olfference  equations  of  the  form 

^  -  au(t)  bu(t  —  1)  +  v(t),  t  >  1 

(!*▼) 

u(t)  -  c(t),  0  <  t  ^  1. 

Problems  of  the  type  we  shall  treat  arise  in  the  study 
of  '’bottleneck"  processes  In  mathematical  economics,  in  the 
"bang— bang"  control  problem  of  electronics,  and  In  prediction 
theory.  We  shall  discuss  the  stochastic  version  of  these 
problems  in  a  subsequent  paper. 

i^2.  Terminal  Control  of  Linear  Systems .  In  order  to  Illus¬ 
trate  the  reduction  In  dimensionality  that  can  be  effected 
by  the  techniques  we  shall  employ,  let  us  consider  the  follow¬ 
ing  problem:  "  Minimize  lu(T)|  over  all  functions  satisfying 
the  constraints  .p 

|v(t)|  <  kj,  0  <  t  <  T.y'  v^(t)dt  ^  k-,  (J.l) 

0 

where  u  and  v  are  connected  by  means  of  the  linear  equation 

u"  +  au'  +  bu  ■  V,  u(0)  -  Cp,  u'(0)  -  Cp.  (2.2) 

Although  many  problems  of  this  type  can  be  resolved  explicitly, 
cf.^,  the  solutions  are  not  simple.  Furthermore,  slight  changes 
in  the  constraints  can  lead  to  failure  of  the  analytic  techniques. 
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UBing  tht  Lagrange  multiplier  technique  described  In 

Me  can  reduce  the  computational  problem  to  one  involving 

sequences  of  two  variables  by  considering  the  new  functional 

T 

J(v)  .  !u{t)|  +J/  (2.5) 

0 

and  employing  the  functional  equation  technique  as  outlined 
above . 

Let  us  now  show  that  we  can  reduce  the  computation  to 

one  involving  sequences  of  functions  of  one  variable.  As  we 

know,  the  solution  of  (2.2)  may  be  written  in  the  form 

t 

u  •  c^u^(t)  +  k(t  -  •)v(B)ds,  (2.4) 

where  u^  and  u^  are  particular  solutions  of  the  homogeneous 
equation  and  k(t)  is  a  known  function  of  t.  The  problem  posed 
above  is  thus  equivalent  to  that  of  minimizing 

J(v)  -  I  a  •¥  k(T-s)  v(s)ds|  v^dt,  (2.5) 

0  0 

where  a  is  a  fixed  constant.  Introduce  the  function  of  a  and 
T  defined  by 

f(a,T)  -  Min  J(v),  (2.6) 

V 

where  |v(t)|  <  kj^,  0  <  t  <  T. 


Assume  that  v  has  been  chosen  over  some  initial  infini¬ 
tesimal  interval  .  Then 

A  T 

f(a,T)  •  la  -  8)v(B)d8  ■♦■j/^k(T  -  s)v(B)d8| 

0  A 

T  T  T— A 

v^dt  ^  v^dt  -  |a  +  k(T)v(0)A  ^  k(T  -A — b)v(8  -►  A)d8|  + 
0  A  0 

«  T  -  A  P 

/^Av(0)‘^  ^^y  V  (s  A)ds  -f  0(a) 

0 


Using  the  principle  of  optimality,  we  see  that  this 
equation  is  equivalent  to 


I 
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f(a,T) 


Min  pAv(O)^  +  f(a  +  kTv(0)A,  T  -A) 
v(0)  L 


♦  (2.d) 


where  | v (0)|  < 


The  conputatlorja I  and  analytic  effort  thUl  Involves  a  se¬ 
quence  of  functions  of  one  variable.  This  dimensionality 
would  be  maintained  even  If  u  eatleflei  a  linear  equation  of 
arbitrary  degree.  The  principal  cf  euperpoaltlon  plays  an 
eaeentlal  role  here. 


.  Terminal  Control  and  Retardation .  Since  the  solution  of 
and  of  linear  equations  of  this  general  class,  can  be 
written  In  the  form^ 

u(t)  «  w(t)  k(t  -  8)v(e)d8,  t  >  1,  (3.1) 

0 

where  w(t)  and  k(t)  are  known  functions, 

terminal  control  problems  Involving  linear  systems  with 
retardation  can  be  treated  In  precisely  the  same  fashion. 

If,  In  place  of  an  expression  such  as  ;u(T)|,  we  wish  to 
minimize  an  expression  of  the  form  g[u(T),  u^(T)],  we  will 
require  sequences  of  functions  of  two  variables. 

Discussion .  Generally,  If  x(t)  Is  governed  by  a  linear 
equation  of  the  form  given  In  (i‘.3),  a  direct  application  of 
the  functional  equation  technique  will  lead  to  functions, 
f(c,T)  of  dimension  N.  If  the  objective  is  to  minimize  or 

9  ^ 

maximize  a  function,  gl_J^j^(T),  x^(T),  ...»  Xj^(T)],  of  the 
first  k  components  of  x(T),  the  metfiOd  sketched  above  will 
lead  to  functions  of  k  variables.  In  a  number  of  applications 
In  the  economic  and  engineering  field,  we  wish  to  control  the 
final  states  of  only  one  or  two  Significant  quantities.  The 
method  outlined  above  furnishes  a  computational  solution  to 
problems  of  this  type. 
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